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METHOD FOR EVALUATION AND STABILIZATION OVER TIME OF 

CLASSIFICATION RESULTS 

Background Of The Invention 

The invention proceeds from a method and system for evaluating 
classification results from classification methods which 
5 proceed in computer-assisted fashion and in which objects to 
be classified are sensed using sensors over a period of time, 
and are repeatedly classified with the inclusion of specific 
quality parameters for an object class. A quality parameter of 
this kind is a parameter number or another qualifying 
10 indication within a preselected range, and is expressive of 

the certainty that the object belongs to the particular object 
class and not to a different object class. 

Observations of objects are evaluated today in many 
15 application sectors, e.g. in machine vision in industrial 
production processes, in safety-related applications, for 
sensing the surroundings of vehicles, etc. These observations 
are furnished by measurement devices that can comprise, in 
principle, one or more sensors that in some cases is 
20 different, e.g. radar, ultrasound, video in the visible and 
infrared region, lidar, laser, range imager. 

One goal when processing object data obtained in this fashion 
is classification. "Classification" is understood to be the 
25 allocation of objects to specific object classes. A class 

serves to describe multiple objects that are considered, on 
the basis of similar features, to belong together. 
Classification systems for the assignment of objects to object 
classes are used, for example, in industrial production. 
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Sensing and object classification of the surroundings of a 
motor vehicle while driving also promises great benefits. 

Classification according to known classification methods has 
5 the disadvantage that, especially in borderline cases with 
regard to allocation of the sensed object to a class, an 
elevated risk of misclassif ication exists. In borderline 
cases, for example, there is a high probability that in the 
context of individual classification results, the object will 
10 be attributed to the wrong object class. Outliers in the 

context of classification of an object, resulting e.g. from 
measurement errors, also have a negative effect on the 
stability and robustness of presently known classification 
methods . 

15 

The methods according to the existing art furthermore offer 
only a yes/no classification of the sensed object. These 
results are insufficient for situations that require a more 
differentiated classification result, for example in the 

20 context of estimating accident hazards for a motor vehicle. 

One reason for this is that in the context of a classification 
repeated over time, an object is allocated first to one class 
and then, as the object is tracked, later on to a different 
class; this instability means that a clearly interpretable 

25 result is not supplied. Information as to how certain it is 
that the object might not also belong to one of the other 
classes would therefore be additionally important. 

Advantages Of The Invention 

30 

The features of the independent claims result in a 
stabilization over time of the classification result. 
According to the present invention, a confidence parameter is 
calculated from the quality parameters of the individual 
35 measurements, and is generally 11 integrated" over a certain 
number of time steps k and corresponding individual 
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measurements. An abruptly large change in the confidence 
parameters for each object class with reference to successive 
time steps is thereby avoided, so that individual outliers 
during classification can be filtered out. A strengthening or 
5 weakening of the classification result is also effected when a 
subsequent classification of the result confirms or does not 
confirm a previous one. "Confirmation" is understood here as 
at least a tendency toward allocation of the object to the 
same class. For example, a classification with reference to an 
10 object class as the third most suitable class among ten 
classes can also represent a confirmation for a previous 
classification of the object in that class. 

Because, according to the invention, the method continuously 
15 repeats, the classification result in borderline cases is 

stabilized to the somewhat more probable object class, so that 
disruptive back-and- forth jumps in the classification result 
are suppressed and the method is robust in terms of such 
disruptions . 

20 

The utilization of time-related information in the context of 
the present invention furthermore makes possible a more 
differentiated evaluation of the classification result. This 
kind of "final classification result," based on time-related 

2 5 context information and as a function of confidence parameters 

adapted by way of many measurements, can yield a substantially 
better result than a classification based on individual 
results . 

3 0 The dependent claims describe advantageous embodiments and 

refinements of, and improvements to, the respective subject 
matter of the invention. 

One advantageous refinement of the method according to the 
3 5 present invention is a dependence of the increase and/or 

decrease in the value of the respective confidence parameter 
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on its absolute quality, the "absolute quality" of the 
confidence parameter being understood as its level. With a 
high confidence parameter, for example, a value increase can 
be correspondingly small so as not to create too great a 
5 difference between the confidence parameters, since too great 
a difference would otherwise interfere with an inherently 
correct correction of a misclassif ication . 

This problem is also solved in another refinement of the 
10 method in which the permissible value range for the confidence 
parameters is limited. 

An additional improvement to the method according to the 
present invention is represented by weighted inclusion of the 

15 absolute quality of the respective individual classification 

results, with reference to the individual object classes, into 
the increase and/or decrease of the corresponding confidence 
parameter. The reliability of the individual results for each 
object class is thus also included in the classification 

20 result. A higher classification quality is the consequence. 

A further advantageous method step is that of evaluative 
analysis of the calculated confidence parameters to determine 
a final, detailed classification result. It is thereby 
25 possible to arrive at more-differentiated statements regarding 
classification, for example the statement that the object fits 
badly into class A and fits very badly into all other classes. 
This is an improvement over the statement that "class A is the 
best fit." 

30 

Also advantageous are on the one hand an evaluation 
constituting a classification into a higher-order class in the 
event the classification results alternate between similar 
object classes (example: if the result alternates between 
35 "small car" and "large car," the final result might be the 

higher-order class "car"), and on the other hand an evaluation 
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constituting a refusal to classify if the classification 
results jump back and forth over time between dissimilar 
object classes (example: if the result alternates between 
"truck" and "pedestrian, " the final result might be "object 
5 cannot be reliably classified") . 

The method according to the present invention can be used, in 
particular, to evaluate the classification results of a 
classification method for objects in the surroundings of a 
10 vehicle. This can be accomplished using a computer-assisted 
vehicle information system that contains the following 
devices : 

- effective connections to relevant interfaces to vehicle 
15 sensor devices for sensing objects in the vehicle's 

surroundings ; 

- a control circuit for analysis and classification of the 
sensed objects, suitable for carrying out the method according 

2 0 to the present invention; and 

- effective connections to relevant interfaces to actuator 
devices on the vehicle. 

2 5 Drawings 

Exemplifying embodiments of the invention will be explained 
with reference to the drawings, in which: 

30 Figure 1 is a block depiction of a computer-assisted 

vehicle information system set up to carry out 
an embodiment of the method according to the 
present invent ion ; 

3 5 Figure 2 is a flow chart for an embodiment of the method 

according to the present invention; and 
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Figure 3 is a sketch of a scene in the surroundings of a 

vehicle, with results from an embodiment of the 
method according to the present invention. 

Description Of Exemplary Embodiments 



In the Figures, identical reference characters denote 
identical or identically functioning components. 

10 

Referring to Figure 1, sensors 5 for sensing objects (e.g. 
trees, pedestrians, passenger cars, or trucks) in the 
surroundings of vehicle 1 are mounted in and/or on vehicle 1, 
for example a passenger car. The nature of sensors 5 can be, 
15 for example, radar, ultrasound, video in the visible and 
infrared region, lidar, laser, and range imager camera. 

The measured data acquired by the sensors regarding a sensed 
object are forwarded via effective connections and interfaces 

20 (e.g. cable, radio, infrared transmission) to vehicle 

information system 3 integrated into vehicle 1. Vehicle 
information system 3 possesses a computer unit 7 associated 
with which is a database 9 that contains data about features 
regarding object classes such as pedestrians, transport 

25 vehicles, and passenger cars. 

Also part of computer unit 7 is a control circuit 11 for 
classifying the sensed objects and evaluating the 
classification result. Both the measured data from sensors 5 

3 0 and the data about the features of the object classes required 
from database 9 are fed into this control circuit 11, and 
processed therein for classification. According to the present 
invention, the confidence parameters, which indicate how well 
the sensed object matches the particular object class, are 

3 5 correspondingly increased or decreased in value depending on 
whether the previous classification is or is not confirmed, 
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e.g. raised by three points per iteration, or lowered by three 
points for a decrease in value. 

According to the invention, a programmed evaluative analysis 
5 of the classification result also occurs in computer unit 7. 
This results in a differentiated classification, for example 
the statement that the object fits very well into class A and 
fits moderately well into all other classes. 

10 When the classification result has achieved sufficient 
stability over time according to the present invention, 
specific actuator devices 13 of vehicle 1 can be triggered by 
control unit 7, if desired, via effective connections (e.g. 
cable, radio, infrared transmission) ; for instance, the brake 

15 can be actuated if the classification result is used, for 
example, in conjunction with a collision prediction and a 
safety system. 

Figure 2 depicts an embodiment of the method according to the 
20 present invention as a flow chart. 

A preclassif ication 100, which is based on a single 
measurement using the aforementioned sensors and a single 
interpretation of the measurement results obtained therefrom, 

25 and does not yet contain any stabilization over time and does 
not yet represent a confidence parameter as defined in this 
Application, supplies, for each object to be investigated, 
parameter numbers constituting a "quality parameter" as 
defined above, for matching to all the classes that are to be 

30 distinguished. Motivated by information theory, what is used 
in this example as the parameter is a so-called word length, 
which should be short in the event of a good agreement between 
object and class. This "classification word length" is 
referred to hereinafter as "w_i," where i is the index for a 

35 class. 
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As a possible enhancement, a "rejection word length" w_rej_i 
can be calculated in addition to the classification word 
lengths w_i . This parameter refers to the question of whether 
the object does NOT fit into the particular object class and 
5 is to be rejected. The calculation of w_rej_i can proceed, in 
principle, differently from that of w__i . Both 
preclassif ication results (w_i and w_rej_i) , or conversely 
only one of the two, are used in this example as input 
variables of the method according to the present invention; 
10 i.e. the aforementioned "quality parameter" serves for 

calculation of the actual confidence parameter, as described 
in more detail below. 

For the object to be classified, a so-called hypothesis exists 
15 for each of the object classes to be distinguished. Hypothesis 
i is "the object belongs to class i." 

The probability that a hypothesis is true is evaluated using 
the confidence parameter "conf_i." The method according to the 
20 present invention uses these confidence parameters to evaluate 
all the classification word lengths. 

In addition to the input data mentioned above, according to 
the present invention the evaluation system uses the 
25 confidence parameters from previous time steps (k-1), i.e. 
"conf_i_k-l , " for each of the individual hypotheses. 

In processing step 110 (adaptation of the confidence 
parameters), a distinction 120 as to specific cases is made. 
3 0 These cases can be handled differently depending on the 

application, i.e. the system can in principle also be used 
without a distinction of cases, or with fewer or more 
distinctions. In this example the following three cases are 
distinguished : 

35 



NY01 1075389 vl 



The object was classified in the current time step, and 
not rejected. 

The object was categorized by the classifier in the 
current time step as belonging to none of the object 
classes, and therefore rejected. This can be done, for 
example, by comparing the rejection word lengths w_rej__i 
for all the classes with a defined rejection threshold. 

No classification word lengths (w_i and/or w_rej_i) exist 
for the object in the current time step. This can be 
attributable to insufficient classifier input data, but 
can also result from deliberate lack of processing of the 
object. It may be useful, for example, in the interest of 
high processing speed, not to reclassify in the current 
time step an object that already has a high confidence 
parameter, but instead to perform the classification only 
on the basis of the old confidence parameters. 

2 0 Case a) - The object was classified in the current time step : 

According to the classification word lengths w_i , an increase 
in the value of the corresponding confidence parameter must be 
performed. One possibility is to increase the value according 
25 to a quality sort of the individual hypotheses: all the word 
lengths w_i are sorted in ascending order, and the smallest 
w_i indicates class that "best matches" the object. The 
confidence parameters for each hypothesis are increased in 
value in accordance with this ranked list. The formula might 

3 0 be as follows: 

confidence parameter for the class in first place on the 
list is increased in value by 5 points; 

3 5 - confidence parameter for the class in second place on the 
list is increased in value by 3 points; 



a) 



b) 



10 c) 



15 
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confidence parameter for the class in third place on the 
list is increased in value by 1 point; 

5 - all other confidence parameters remain unchanged. 

The number of classes to be taken into account, and the 
relationship among the value increases, can be selected 
without restriction based on the particular application. 

10 

The increase in value is additionally performed as a function 
of the absolute quality of the individual result. Here an 
additional weighting factor g_i is calculated for each class 
(see step 130) . This factor takes into account the quality of 
15 the result for each class (i.e. the level of w_i in relation 
to a specific threshold) . The distance from w_rej_i to the 
rejection threshold may be used here as an example: this 
distance is an indication of the evaluation "the object does 
not belong to any of the classes." 

2 0 

g_i = max (0; 1 - (w_rej_i / rejection threshold)) 

For objects that were classified but almost rejected, the 
weighting factor assumes a value close to 0. This means that 

25 the corresponding hypothesis was in fact barely confirmed, and 
an increase in the value of the confidence parameter is 
therefore not plausible. For a higher-quality result 
(corresponding to a greater distance from the rejection 
threshold), g_i approaches 1. This applies to the assumption 

30 of a positive w__rej_i. 

The result for the increase in value of the confidence 
parameters is then: 5 * g_i for the class in first place, and 
3 * g_i and 1 * g_i correspondingly for second and third 
35 place, respectively. 
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Case b) - The object was "rejected" by the classifier in the 



current time step : 



In the event of rejection, the confidence parameters of all 
5 the hypotheses are decreased in value: a rejection of the 

object means that the object cannot be allocated to any of the 
defined classes, and therefore none of the defined hypotheses 
can be supported. 

10 Correlated with the quantitative example of a 5, 3, 1 value 

increase cited in case a) , a value decrease of, for example, 4 
for all confidence parameters can be applied in case b) . 

As in case a) , a quality- dependent additional weighting factor 
15 g2_i is used for the value decrease in the confidence 

parameters for each class i (130) . This factor can once again, 
for example, be based on the distance between the rejection 
word length w_rej_i and a defined rejection threshold: 

20 g2_i = min (1; [1 - (w_rej_i / rejection threshold)]) 

In this case g2_i assumes large values (a maximum of 1) for a 
large distance from the rejection threshold: the more 
unequivocally the object was rejected, the more the value of 
25 the confidence parameter must be decreased. For a w_rej_i 
close to the rejection threshold, weighting factor g2_i 
approaches 0 (no value decrease) . 

Case c) - No classification result exists for the object in 
3 0 the current time step : 

If the classification word lengths (w_i and/or w_rej_i) are 
absent, all the confidence parameters are decreased in value 
by a specific identical amount. This represents a reduction in 
35 the certainty that the hypotheses are correct. For the 

quantitative example being used, a decrease in the value of 
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all confidence parameters by 1 is selected. It is obvious in 
this context that the absolute numbers should be coordinated 
with the classification iteration rate, the value range 
limits, and the application of the method. The iteration rate 
5 here is the frequency with which individual quality parameters 
are calculated, per predefined unit of time, in the aforesaid 
time steps k. 

Here as well, it is advantageous to apply a more 
10 differentiated value decrease as a function of various cases 
(e.g. by asking "why was the object not classified," etc.). 

Adapting the value range (see step 140) : 

15 An adaptation 14 0 of the value range of the confidence 

parameters is effected by limiting them. Here as well, it is 
obvious that the absolute numbers should be coordinated with 
the classification iteration rate and the application of the 
method. 

20 

The value range does not necessarily have to be symmetrical 
about zero; a smaller range for negative confidence parameters 
and a larger range for positive ones may prove useful 
depending on the application. In this example a range from -20 
25 to 200 is selected. 

If the confidence parameters depart from this value range as a 
consequence of an increase or decrease in value, an adaptation 
is necessary; otherwise a classification would no longer be 
30 possible, since all (or at least several) of the hypotheses 
would be supported to the same extent. This can be done 
differently for a value range overflow and underflow. One 
possible adaptation is: 

35 A value range underflow causes the confidence values to 

be truncated. If a confidence parameter is on the verge 
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of dropping below this level, it is set to the minimum 
possible value. 

A value range overflow causes the confidence parameters 
5 not to be simply truncated, but to be adapted in 

accordance with their relationship to one another. When a 
confidence parameter reaches the maximum value, for 
example, a value decrease can be performed for all the 
other confidence parameters by the same amount by which 
10 the high confidence parameter would normally be increased 

in value. The distance between the high confidence 
parameters and the other confidence parameters is thereby 
maintained. 

15 The aforesaid adaptation is only an example of the plurality 
of possible procedures. 

Determining the final classification result (step 150) : 

2 0 According to the present invention, a determination 150 of the 
classification result for the object in question is performed 
as the last processing step of the method according to the 
exemplifying embodiment. This result is ascertained on the 
basis of the calculated confidence parameters. 

25 

A very large number of possibilities exist here for detailed 
evaluation, for example: 

Assign the object to the class for which the highest 
30 confidence parameter exists. 

As an additional alternative: the classification result 
is valid only if this highest confidence parameter 
exceeds a given minimum threshold. 

35 
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As an additional alternative: the classification result 
is valid only if the distance between this highest 
confidence parameter and the second-highest confidence 
parameter exceeds a defined threshold (otherwise the 
5 result is characterized as "not unequivocal") . 

For stabilization over time according to the present 
invention, the confidence parameters are temporarily stored 
and are used as input data for the next processing step of the 
10 method. 

Figure 3 depicts a scene in the surroundings of a vehicle, 
showing the results of an embodiment of the method according 
to the present invention. For each of the objects present in 
15 the scene (vehicles only, in this case) , one of which is 

labeled with reference character 206, icons 200 show the class 
of the object by way of a symbol 2 02, and indicate the level 
of the confidence parameters using a bar 204. This depiction 
is not intended for the driver, but serves only for further 

2 0 explanation of the present invention. 

Although the present invention has been described above on the 
basis of a preferred exemplifying embodiment, it is not 
limited thereto but rather is modifiable in numerous ways, 
25 e.g. the absolute magnitude of the value increase/decrease 
delta numbers, the iteration duration, or the limits. 

The method according to the present invention can be used to 
improve any type of classification. The method encompasses a 
30 plurality of steps and variants that for the most part are 
mutually independent, and it can be flexibly configured. 

Lastly, the features of the dependent claims can be combined 
with one another substantially without restriction and not in 

3 5 the sequence existing in the Claims, provided they are 

independent of one another. 
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